OPENNLP-1866: Update Dev Manual on Unicode normalization and the UAX #29 tokenizer (4-docs/7) - #1106

OPENNLP-1850 stacked PRs (review independently; merge bottom-up, re-targeting each base to main as the one below lands):

1. OPENNLP-1850: Unicode normalization foundation — CharClass engine, rungs, Dimension (1/4) #1103 — Unicode normalization foundation (CharClass engine, rungs, Dimension)
2. OPENNLP-1850: UAX #29 word tokenizer and the layered Term model (2/4) #1104 — UAX OPENNLP-910: Add checkstyle #29 word tokenizer + layered Term model
3. OPENNLP-1865: Support offset-safe input normalization in the DL components (3-dl/7) #1105 — Offset-safe input normalization in the DL components
4. OPENNLP-1866: Update Dev Manual on Unicode normalization and the UAX #29 tokenizer (4-docs/7) #1106 — Documentation

Supersedes #1101.

Thx for the PR. Here are some suggestions:

• Declare xmlns:xlink explicitly on the chapter roots of normalizer.xml and tokenizer.xml. Both use xlink:href (normalizer.xml:457, tokenizer.xml:461) but rely on the DocBook 5.0 DTD's #FIXED default to bind the prefix. The Maven build resolves the DTD so it works, but it breaks under any non-validating namespace-aware tool (IDE linters, xmllint --nonet), and every other chapter declares it explicitly:
  normalizer.xml: <chapter xml:id="tools.normalizer" xmlns:xlink="http://www.w3.org/1999/xlink">
  tokenizer.xml: <chapter xml:id="tools.tokenizer" xmlns:xlink="http://www.w3.org/1999/xlink">
  Note that tokenizer.xml newly introduces xlink usage, so this is the first chapter to add it there.

Otherwise the content is accurate.

Status since the last review. Normalizer chapter gains an "Offset-aware pipelines" section for buildAligned() and the capability interface with a worked dash-fold span example, the line-break-preserving rung in the fold table, and the supplementary-dash offset note in the DL fold options. Name-finder chapter names OffsetMappingNameFinder behind findInOriginal. docbkx HTML builds clean. Rebased onto the updated stack.

Hi @krickert. I didn't have time to deeply review yet, so here you get some Fable 5 output. Take it with a grain of salt — I will read it next week myself (human in the loop).

The good news first: the model cross-checked the docs against the actual code on the branch and found the technical content fully accurate — all constructors/signatures in the samples exist as shown, all 12 xref targets resolve, XML is well-formed with the header matching the post-OPENNLP-1854 DTD convention, defaultChain() order matches the prose, and every expected-output comment ("cafe", the span arithmetic, 3.0 : NUMERIC, …) is pinned by an existing unit test. Everything below is clarity/duplication/structure, no factual errors.

Findings (most severe first):

1. Duplication with drift (namefinder.xml, doccat.xml, normalizer.xml): the InferenceOptions folding + White_Space chunking story is told near-verbatim in three chapters, and the copies already disagree in wording ("folds each Unicode whitespace character" vs. "maps each Unicode whitespace code point" vs. "one code point to one space"). Suggestion: component chapters keep one sentence + the existing xref to tools.normalizer.dl as the single canonical explanation.

2. tokenizer.xml intro: "The DL components apply that machinery automatically for document chunking" reads as if DL normalization is automatic, contradicting the "off by default" statements elsewhere — only chunking is automatic. The new paragraph also splits the intro sentence from the <screen> example it introduces. Suggested rewording: "The DL components perform Unicode-aware chunking automatically; the folding transforms remain opt-in."

3. doccat.xml: the supplementary-plane dash / UTF-16 offset caveat is spelled out in full, but DocumentCategorizerDL exposes no span/offset API, so readers get surrogate-pair arithmetic they can never observe. One clause + xref would do.

4. namefinder.xml: the second code sample repeats ~14 lines of the first (model, vocab, full 9-entry ids2Labels map, sentence detector) to add three InferenceOptions lines — and the copies already drift (different tokens arrays, different comments). Could be trimmed to just the options + constructor call.

5. normalizer.xml: "rungs" is used as terminology three times but never introduced, while the same file also says "stage" for the same concept. Pick one (plain "stage" reads fine).

6. namefinder.xml/doccat.xml: the offset-preservation sentence ("Whitespace folding is one code point to one character …, and so is dash folding for Basic Multilingual Plane dashes; a supplementary-plane dash shrinks from two UTF-16 units to one …") is a garden path with mixed units and unglossed jargon. Splitting into three short sentences (whitespace always safe → BMP dashes safe → rare non-BMP dashes shift, findInOriginal compensates) would fix it.

7. normalizer.xml offset-aware section: two very dense sentences — the buildAligned() description with three nested "returns", and a ~63-word sentence ending "an offset that does not hold". Both accurate, both worth splitting.

8. normalizer.xml intro: the "three principles" describe implementation strategy ("cursor-based, no regular expressions", "O(1)", "ReDoS"), and this internals-talk recurs ~8 more times across three chapters. The user-relevant contract is "Unicode-correct per the UCD, offset-preserving" — the no-regex/performance story could be stated once (or left to Javadoc) so the manual doesn't freeze internals as promises.

Minor: "lists the 25 characters carrying the White_Space property" will silently go stale on a UCD upgrade; the Catalan middle dot exemption in the bullet-normalizer table would benefit from the one-word reason (U+00B7 occurs inside Catalan words); "adding an unrequested dimension costs one transform" reads as a contradiction until you realize "adding" means "querying at runtime".

@rzo1 addressed all the points in your claude. I'll have copilot do a round as well - cant hurt.

Thanks for improving the docs.

Thanks for improving the docs.

Lots more coming.

Something looks weird here. There are commits + code changes in a PR that claims to provide the documentation. @krickert Can you verify the branch is up2date with main (-> rebase) and provide it in a clean fashion?

On it let me reproduce it locally and take a look.. Should be pretty quick to do

I had to rebase onto current main after #1105 merged. The branch is now 12 commits, 6 docbook files only (+778/−8). The earlier Java/DL noise was duplicate stack history. It should be gone after the reset/cherry-pick. Sorry about that, that one really did look weird.